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SUMMARY 


The  Advanced  On-the-job  Training  System  (AOTS)  was  an 
Air  Staff-directed,  AFHRL-deve loped,  prototype  system  which 
designed,  developed,  and  tested  a  proof-of -concept  prototype 
AOTS  within  the  operational  environment  of  selected  workcenters 
at  Bergstrom  AFB,  TX  and  Ellington  ANGB,  TX  from  August 
1985  through  31  July  1989.  This  performance  study  report 
is  a  part  of  the  System-Level  Testing  and  Evaluation  of 
the  AOTS.  This  study  investigated  the  impact  of 

Computer-Based  Training  (CBT)  modules  on  Air  Force  maintenance 
task  knowledge  and  task  performance  by  comparing  airmen 
who  reviewed  the  CBT  modules  (treatment  group)  with  a 
comparable  group  of  airmen  who  did  not  review  the  modules 
(control  group) .  Findings  from  this  study  showed  that 

airmen  receiving  CBT  scored  significantly  higher  on  a 
majority  of  tasks  in  terms  of  knowledge  and  also  scored 
higher  on  two  of  the  seven  tasks  in  terms  of  performance. 
Findings  from  structured  interviews  with  treatment  subjects 
on  the  efficacy  of  CBT  for  Air  Force  maintenance  training 
are  also  reported. 
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PREFACE 


This  paper  documents  a  performance  study  conducted  to 
investigate  the  impact  of  Computer-Based  Training  modules 
on  Air  Force  maintenance  task  training.  As  a  part  to  the 
System-Level  Testing  of  the  Advanced  On-the-job  Training 
System  (AOTS),  it  investigated  differences  in  task  knowledge 
and  task  performance  by  comparing  airmen  who  reviewed  CBT 
modules  with  a  control,  group  of  airmen  who  received 
traditional  on-the-job  training  (OJT).  The  AFHRL  work  unit 
number  for  the  project  is  2557-00-02.  The  Air  Force  AOTS 
manager  is  Major  Jack  Blackhurst. 
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ADVANCED  ON-THE-JOB  TRAINING  SYSTEM  (AOTS) 
PERFORMANCE  STUDY 


Introduction 

Computer  Based  Training  (CBT)  is  considered  a  viable  and 
cost-effective  training  tool.  After  the  initial  cost  of 
development,  CBT  can  be  used  continuously  to  provide  standardized 
training.  It  can  supplement  training  conducted  by  trainers, 
reducing  the  need  to  depend  on  equipment  or  personnel.  In 
addition,  the  audio-visual  presentation  mode  has  advantages  over 
written  instruction  manuals.  All  these  features  add  to  the 
potential  of  CBT  to  improve  training  in  the  operational  setting. 

The  past  decade  has  seen  a  proliferation  of  research  on  the 
efficacy  of  CBT  (Clements,  1986?  Dalgaard,  1984;  Kulik,  1983; 
Kulik,  Bangert,  &  Williams,  1983;  Kulik  &  Bangert-Drowns ,  1984; 
Kulik  &  Kulik,  1987).  Research  findings  on  the  effectiveness  of 
these  technologies  to  enhance  student  achievement  are  often 
mixed,  ranging  from  little  or  no  effect  to  substantial  gains. 
Kulik  (1983),  however,  argued  that  several  findings  were  now 
clear.  Based  on  two  meta-analysis  reviews  conducted  at  the 
University  of  Michigan,  he  concluded  that  Computer  Aided 
Instruction  (CAI) :  1)  has  the  potential  to  improve  student 
achievement;  2)  fosters  positive  attitudes  toward  computers;  and 
3)  reduces  learning  time.  These  conclusions  were  reaffirmed  by 
Kulik  &  Kulik  (1987)  in  a  separate  meta-analysis  of  199  CBT 
studies  conducted  at  the  primary,  secondary,  and  university 
levels. 

Kearsley  (1983)  argued  that  CBT  can  improve  training 
effectiveness  because  of  the  high  level  of  feedback  and  practice 
it  provides.  Feedback,  according  to  Kearsley,  ensures  that 
students  understand  the  materials  being  taught  while  practice 
provides  students  with  an  opportunity  to  increase  their  skill 
levels  and  retention  rates.  He  warns  us,  however,  that  CBT  is 
unlikely  to  have  a  significant  effect  on  job  performance  or 
training  effectiveness  if  the  CBT  courses  themselves  are  poorly 
developed. 

Although  a  major  use  of  Computer  Based  Training  has  been  in 
technical  schools,  application  of  this  technology  in  the 
operational  job  setting  has  generated  a  substantial  amount  of 
interest.  Computer  Assisted  Instruction  and  Interactive  Video 
Disk  technology,  for  example,  can  enhance  the  acquisition  of 
knowledge  and  facilitate  task  performance  in  the  operational 
setting.  Previous  research  on  the  efficacy  of  this  technology 
has  focused  on  the  effects  of  computer-based  training  lessons 
with  minimal  emphasis  on  the  supplemental  effects  of  CBT  on 
hands-on  training.  The  portability  of  this  technology  to  the 
work  setting,  however,  makes  it  possible  to  investigate  the 
supplementary  effects  that  CBT  modules  have  on  hands-on  training. 
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Although  AF  task  training  is  conducted  by  using  a  technical 
manual,  or  Technical  Order  (T.O.),  most  maintenance  training 
lacks  standardization.  The  majority  of  task  training  is 
conducted  on  an  apprenticeship  basis:  over-the-shoulder,  learning 
by  doing,  and  with  occasional  reference  to  the  T.O..  While  this 
method  is  the  accepted  mode  of  AF  On-the-job  training  (OJT) ,  this 
study  is  primarily  focused  on  the  supplemental  effects  of  CBT  on 
hands-on  training.  The  question  of  interest  to  this  study  is  — 
What  are  the  effects  of  CBT  on  the  task  knowledge  and  performance 
of  Air  Force  (AF)  personnel  who  have  already  been  certified  on 
the  particular  task  or  tasks  in  question?  Certification  on  a 
task  is  a  formal  indication  that  an  individual  has  the  ability  to 
perform  that  task  (AFR  50-23) . 

Mefchgd 

Subjects.  The  majority  of  the  participants  in  this  study 
were  first -term  airmen  from  two  Air  Force  specialties:  Jet  Engine 
Mechanic  (JEM)  and  Aircraft  Maintenance  for  the  F-4  aircraft. 
There  were  24  subjects  from  the  67th  Tactical  Reconnaissance  Wing 
(TRW) ,  composed  of  12  treatment  subjects  from  the  12th  Aircraft 
Maintenance  Unit  (AMU)  and  an  equal  number  of  control  subjects 
from  the  91st  AMU.  Because  of  the  limited  number  of  JEM 
personnel  at  Bergstrom  AFB,  the  treatment  and  control  groups  were 
from  2  different  bases.  Bergstrom  AFB  (67th  Aircraft  Generation 
Squadron  [AGS])  provided  ten  treatment  subjects  and  ten  AF 
personnel  from  a  comparable  F-4  base,  George  AFB,  California 
( 35th/37th  AGS),  were  used  as  control  subjects.  Subjects  in  both 
the  treatment  and  control  groups  were  selected  on  the  basis  of 
two  criteria:  time  in  service,  and  recency  of  certification 
(within  the  past  2  years)  on  the  tasks  of  interest  to  this  study. 
Because  of  the  limited  availability  of  personnel,  the  criteria 
for  selection  were  not  completely  satisfied,  requiring  a  closer 
examination  of  the  background  characteristics  of  both  groups. 

A  comparison  of  the  participants  did  not  reveal  any 
significant  differences  in  several  key  areas:  time  in  service, 
rank,  number  of  times  tasks  were  performed,  and  last  time  (in 
weeks)  each  task  was  performed.  Table  1  shows  a  comparison  of 
demographic  characteristics  (time  in  service  and  rank)  for  both 
the  AMU  and  JEM  groups.  An  analysis  of  variance  for  time  in 
service  (months)  was  performed.  The  ANOVA  showed  no  significant 
difference  in  time  in  service  for  the  treatment  and  control 
groups  within  either  the  AMU  or  JEM  samples.  As  Table  1  clearly 
shows,  the  treatment  and  control  groups  for  the  AMU  sample  were 
similarly  distributed  with  regard  to  rank.  A  non-parametric 
test,  Fischer’s  Exact  Test  of  Association,  performed  on  the  JEM 
sample  showed  no  significant  differences  in  distribution  of  ranks 

(p  <  .08) . 

Tables  2  and  3  show  a  comparison  of  the  control  and 
treatment  groups  based  on  the  number  of  times  a  task  was 
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performed  and  the  last  time  a  task  was  performed.  Based  on  the 
subjects1  self-report  of  each  of  these  variables,  a  univariate 
analysis  of  variance  was  performed.  This  analysis  showed  that 
the  treatment  and  control  groups  (both  AMU  and  JEM)  did  not 
differ  on  either  the  number  of  times  a  particular  task  was 
performed  or  the  number  of  weeks  since  last  performance.  Thus, 
overall  these  two  groups  appear  to  be  equivalent.  Based  on  this 
apparent  equivalence  and  our  knowledge  of  the  AF  personnel 
assignment  process,  it  was  assumed,  for  the  purposes  of  this 
study,  that  AMU  and  JEM  personnel  had  been  randomly  assigned  to 
their  respective  units. 
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Table  1 


Comparison  Of  Demographic  Data  For  JEM  and  AMU  Treatment  And 
Control  Groups  -  APTS  Performance  Study 


AIRCRAFT  MAINTENANCE  JET  ENGINE  MECHANIC 


TIME  IN 

SERVICE 

(months) 

a 

Treatment 

b 

Control 

c 

Treatment 

b 

Control 

Mean 

51.10 

45.36 

52.22 

28.30 

SD 

10.00 

11.00 

9.00 

10.00 

F 

RANK 

.  193 

Treatment 

Control 

1. 

Treatment 

288 

Control 

Amn-SrA 

66% 

66% 

66% 

100% 

Sgt-SSgt 

34% 

34% 

34% 

0% 

Note.  The  control  group  for  JEM  came  from  George  AFB.  All  other 
treatment  and  control  groups  came  from  Bergstrom  AFB. 

a  be 


Q  =  11. 
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n  -  9. 

in 
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** 

B  < 
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Table  2 

Comparison  Of  Task  Relevant  Information  For  JEM  Treatment  And 
Control  Groups 


WEEKS  SINCE  LAST  PERFORMANCE 


TASK 


L10010  L10012  L10017 


a 

TREATMENT 

Mean 

15.89 

17.89 

16.33 

SD 

31.65 

31.08 

31.44 

b 

CONTROL 

Mean 

10.70 

11.00 

8.90 

SD 

19.45 

19.41 

19.20 

F 

.190 

.343 

.396 

NUMBER  OF 

TIMES  PERFORMED 

L10010 

L10012 

L10017 

a 

TREATMENT 

Mean 

7.44 

7.56 

9.87 

SD 

6.38 

6.25 

8.47 

b 

CONTROL 

Mean 

12.60 

12 .30 

18.90 

SD 

8.29 

7.39 

11.22 

F 

2.20 

2.25 

3.93 

a 

b 

n  -  9. 

D  =■ 

10 

* 

** 

in 

o 

• 

V 

Ql 

E 

<  .01 
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Table  3 


Comparison  Of  Task  Relevant  Information  For  AMU  Treatment  And 
Control  Groups 


WEEKS  SINCE  LAST  PERFORMANCE 


110308 

TASK 

J10008 

J10032 

J10159 

a 

TREATMENT 

Mean 

1.10 

16.70 

5.20 

5.20 

SD 

0.32 

25.23 

3.85 

3.85 

b 

CONTROL 

Mean 

1.00 

36.27 

2.64 

2.64 

SD 

0.00 

40.47 

2.11 

2.11 

F 

1.00 

1.74 

3.67 

3.67 

110308 

NUMBER  OF  TIMES 

J10008 

PERFORMED 

J10032 

J10159 

a 

TREATMENT 

Mean 

162.90 

1.10 

9.40 

9.40 

SD 

212.26 

1.60 

8.47 

8.47 

b 

CONTROL 

Mean 

404.36 

9.27 

96.27 

93.54 

SD 

359.85 

12.78 

139.98 

140.83 

F 

3.41 

4.01 

3.82 

3.54 

a 


n  - 


11. 


b 

n  *  10. 


*  ** 
E  <  .  05 


E  <  •  01 
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Procedure .  A  posttest-only  control  group  design  (Campbell  & 
Stanley,  1963) ,  depicted  in  Figure  1,  was  used  to  investigate  the 
effects  of  CBT.  Our  decision  to  use  the  posttest  only  design  was 
based  on  the  assumption  that  AMU  and  JEM  personnel  were  randomly 
assigned  to  their  respective  units  by  the  AF  personnel  assignment 
system.  Randomization  ensures  probabilistic  equivalence  of  the 
treatment  and  control  groups  thus  eliminating  the  need  for  a 
pretest  (Campbell  and  Stanley,  1963;  Campbell  and  Cook,  1979). 

The  apparent  equivalence  of  the  two  groups  as  demonstrated  in 
Tables  1,  2,  and  3  supported  the  assumption  of  randomization. 

Both  kr  -?wledge  and  performance  tests  were  used  to  measure  the 
effects  attributed  to  viewing  the  CBT. 


CBT 

Knowledge 

Performance 

Test 

Test 

Treatment 

Yes 

Yes 

Yes 

Control 

No 

Yes 

Yes 

Figure  1.  Posttest-only  control  group  design 


Four  AMU  and  three  JEM  maintenance  tasks  were  used  during 
this  study.  Computer  training  lessons  were  developed  for  these 
specific  tasks  because  they  are  high  priority  training  tasks, 
typically  performed  in  the  operational  setting.  Of  the  4  AMU 
tasks,  2  involved  Computer  Assisted  Instruction  (CAI)  and  2  were 
Interactive  Video  Disks  (IVD) .  Only  one  of  the  JEM  training 
modules  is  an  IVD  lesson  (or  module) . 

CAI  is  a  programmed  lesson  whose  pace  is  controlled  by  the 
viewer  via  a  keyboard.  The  training  lesson  involves  a  set  of 
sequenced  steps  presented  on  the  video  monitor,  with  graphic 
illustrations  to  accentuate  key  points.  The  IVD,  however,  is  a 
video  disc  with  a  description  of  the  task  as  it  is  performed. 

The  audio-video  presentation  is  highlighted  by  close-ups  or 
graphics  to  emphasize  key  aspects  of  the  training  lesson.  The 
viewer  interacts  with  the  presentation  by  touching  the  response 
options  on  the  video  monitor  to  answer  questions  presented  in  the 
lesson. 
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The  following  are  the  identified  AMU  and  JEM  CAI/IVD  modules: 


AMU  CAI 

Modules : 

AMU  IVD 

MODULES : 

110308 

Install  Drag  Chute 

J10008 

Service  Main  Landing 

J10032 

Remove  Main  Landing 

Gear  Strut 

Gear  Wheel/Tire 

J10159 

Install  Main  Landing 
Gear  Wheel/Tire 

JEM  CAI 

Modules: 

JEM  IVD 

Module: 

L10017 

Close  CSD  Drive  Doors 

L10010 

Install  CSD  Generator 

L10012 

Install  CSD/Generator 
Dome  Assembly 

Assembly 

The  CAI  and  IVD  modules  were  viewed  by  the  treatment  groups 
at  the  Human  Resources  Lab  (Bldg.  1808,  Bergstrom  AFB,  TX) . 

After  viewing  the  training  module,  each  subject  completed  a 
knowledge  test.  Control  group  subjects,  however,  completed  the 
knowledge  tests  without  viewing  the  CBT.  With  the  exception  of 
the  control  group  at  George  AFB,  all  of  the  knowledge  and 
performance  evaluations  were  conducted  at  Bergstrom  AFB. 

All  performance  evaluations  were  administered  under 
controlled  conditions  using  performance  evaluation  checklists 
(PECs) .  That  is,  the  JEM  performance  tasks  were  completed  at  the 
Field  Training  Detachment  lab  and  the  AMU  performance  evaluations 
were  conducted  in  a  hangar,  using  a  dedicated  F-4  aircraft.  All 
of  the  performance  evaluations  were  administered  by  the  Universal 
Energy  Systems  (UES) ,  Inc.  subcontractors  —  two  retired  AF 
personnel  with  maintenance  experience  in  the  AF  specialties  of 
interest  to  this  study. 

The  knowledge  tests  (KTs)  and  performance  evaluation 
checklists  (PECs)  used  in  this  study  were  products  of  the  AOTS 
Evaluation  Subsystem.  The  KTs  were  multiple  choice  tests  based 
on  task  proficiency  knowledge  objectives.  PECs  were  used  to 
measure  hands-on  job  performance.  The  PEC  is  composed  of  those 
performance  objectives  whose  completion  constitutes  the 
successful  completion  of  a  task.  The  examiner  administers  the 
PEC  by  checking  pass  or  fail  on  each  task  objective  during  an 
over-the-shoulder  evaluation,  (see  Hedge  &  Teachout,  1986) 

JEM  and  AMU  Subject  Matter  Experts  (SME)  from  the  AOTS 
Instructional  Systems  Team  provided  logistical  assistance  during 
performance  evaluations.  They  scheduled  evaluations,  procured 
maintenance  tools,  and  established  the  inter-rater  reliability 
(percent  of  agreement  between  UES  evaluator  and  SME  on 
performance  evaluation  checklist) .  Inter-rater  reliability  was 
achieved  at  the  90  percent  level  for  all  tasks. 
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In  addition  to  the  knowledge  and  performance  tests, 
individual  interviews  were  conducted  with  15  treatment  subjects. 
The  structured  interview  was  composed  of  multiple  choice  and  open 
ended  questions.  Interview  questions  involved  the  user- 
friendliness  of  the  CBT,  comparison  with  AF  OJT,  and  efficacy  of 
implementing  this  technology  in  the  operational  setting.  The 
administration  time  for  the  16  interview  questions  was  10  to  15 
minutes  (Appendix  A) . 


9 


Results 


Knowledge  and  Performance  Tests.  Tables  4  and  5  contain  the 
means  and  standard  deviations  of  the  percentage  scores  for  each 
knowledge  and  performance  test  broken  down  by  test  group  (i.e., 
JEM  and  AMU) .  Scores  were  calculated  simply  by  using  the  number 
of  correct  responses  per  test  divided  by  the  total  number  of 
items  per  test. 

A  univariate  analysis  of  variance  (ANOVA)  was  performed  for 
the  knowledge  and  performance  tests  associated  with  each  of  the 
tasks.  The  purpose  of  this  analysis  was  to  determine  whether 
differences  in  knowledge  or  performance  existed  between  the 
treatment  and  control  groups  with  regard  to  each  of  the  tasks. 

The  ANOVA  revealed  a  significant  difference  (p  c.Ol)  in  knowledge 
test  scores  for  two  (Service  Main  Landing  Gear  Strut  and  Install 
Main  Landing  Gear  Wheel/Tire)  of  the  four  AMU  tasks.  Here,  the 
group  receiving  CBT  scored  higher  than  the  group  not  receiving 
CBT  on  two  tasks  and  at  a  comparable  level  on  the  other  two 
tasks.  In  the  JEM  sample,  the  treatment  group  scored 
significantly  higher  on  all  three  knowledge  tests  (p  <.oi  for 
Install  CSD  Generator  Assembly  and  Install  CSD/Generator  Dome 
Assembly;  p  <  .05  for  Close  CSD  Drive  Doors).  A  multivariate 
analysis  of  variance  (MANOVA)  showed  a  significant  difference  (p 
<  .01)  between  the  treatment  and  control  groups  in  the  JEM  sample 
for  knowledge  tests.  A  MANOVA  was  not  performed  for  the  AMU 
sample  because  the  associated  variance-covariance  matrix  was  not 
homogeneous . 

Although  there  were  significant  differences  on  most  of  tasks 
with  regard  to  knowledge  tests,  this  finding  was  not  as  prevalent 
for  performance  test  scores.  Within  the  AMU,  three  of  the  four 
performance  scores  were  the  same  for  both  groups.  The  treatment 
group  did,  however,  score  significantly  higher  (p  <.05)  than  the 
control  group  on  one  task  ~  Install  the  Wheel  and  Tire  Assembly. 
Looking  at  the  JEM  specialty,  the  treatment  group  scored  at  a 
comparable  level  with  the  control  group  on  two  of  the  tasks 
(Close  CSD  Doors,  Install  CSD  Generator  Dome  Assembly) ,  and 
significantly  (p  <.05)  higher  than  the  control  group  on  the 
remaining  task  (Install  CSD  Generator  Assembly) . 
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Table  4 


Knowledge  and  Performance  Test  Scores  For  JEM 


L10010 

KNOWLEDGE  TEST 

TASK 

L10012 

L10017 

a 

TREATMENT 

Mean 

84.56 

95.90 

87.03 

SD 

11.03 

5.12 

16.20 

b 

CONTROL 

Mean 

66.66 

67.36 

65.00 

SD 

11.11 

13.99 

21.44 

F 

** 

12.38 

** 

33.28 

* 

6.27 

PERFORMANCE  TEST 


L10010 

L10012 

L10017 

a 

TREATMENT 

Mean 

84.68 

79.32 

100.00 

SD 

11.70 

21.86 

0.00 

b 

CONTROL 

Mean 

95.94 

89.76 

91.43 

SD 

6.01 

8.29 

18.07 

F 

* 

7.19 

1.98 

2.07 

a 


n  - 


9. 


b 

n  -  10 


*  ** 

E  <  .05  fi  <  *01 
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Table  5 


Knowledge  and_  Performance  Test  Scores  For  AMU 

KNOWLEDGE  TEST 


110308 

TASK 

J10008 

J10032 

J10159 

a 

TREATMENT 

Mean 

90.00 

84.49 

55.99 

90.41 

SD 

25.32 

7.97 

20.66 

10.03 

b 

CONTROL 

Mean 

81.81 

68.17 

59.99 

78.78 

SD 

7.04 

14.01 

23.67 

8.43 

F 

1.05 

** 

10.45 

0.17 

** 

8.33 

110308 

PERFORMANCE 

J10008 

TEST 

J10032 

J10159 

a 

TREATMENT 

Mean 

96.66 

83.33 

97.50 

98.32 

SD 

5.67 

26.68 

5.27 

2.68 

b 

CONTROL 

Mean 

100.00 

83.98 

94.32 

88.32 

SD 

0.00 

13.17 

10.25 

11.24 

F 

3.83 

0.01 

0.77 

* 

7.51 

a  b 

n  -  ii.  n  -  10. 


*  ** 

B  <  .05  B  <  »01 
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Pissassigp 


The  results  of  this  study  support  the  use  of  CBT  in  the 
operational  job  setting.  The  knowledge,  performance  tests  and 
interview  results  reveal  benefits  of  computer  based  training 
lessons,  even  with  AF  personnel  previously  certified  on 
maintenance  tasks. 

Higher  knowledge  test  scores  among  the  treatment  groups  were 
expected.  Because  the  knowledge  tests  are  based  on  the  computer 
training  lessons,  it  is  not  surprising  that  the  treatment  group 
earned  higher  scores.  However,  the  finding  that  the  majority  of 
knowledge  tests  were  found  to  be  significant  across  both 
specialties  is  noteworthy.  It  is  noteworthy  because  all 
participants  had  been  previously  certified  on  these  tasks. 
Nonetheless,  the  control  groups  scored  substantially  lower  than 
their  counterparts.  The  higher  scores  on  the  knowledge  tests 
were  greatest  on  the  most  complex  tasks  in  both  the  AMU  (Service 
Main  Landing  Gear  Strut)  and  JEM  (Install  CSD  Generator  Assembly 
and  Install  CSD  Generator  Dome  Assembly) .  Unlike  the  knowledge 
test  results,  the  outcome  of  the  performance  tests  was  not  as 
compelling. 

The  treatment  and  control  groups  scored  at  a  comparable 
level  on  the  majority  of  performance  tests  for  both  the  JEM  and 
AMU.  The  treatment  groups  did,  however,  perform  significantly 
better  on  two  of  the  tasks,  one  from  each  specialty.  Given  this 
finding,  it  is  conceivable  that  the  use  of  CBT  may  accelerate  the 
learning  of  maintenance  tasks.  CBT  may  also  be  useful  as  a 
refresher  course  to  enhance  knowledge  and  performance 
proficiency.  Some  observations  during  the  study  reveal  the 
potential  benefits  of  using  CBT  to  supplement  the  training 
material  in  the  Technical  Order  (T.O.). 

The  participants'  use  of  the  T.O.  during  the  performance 
evaluation  was  informative.  According  to  the  respondents,  the 
T.O.  is  not  as  well  organized  and  thorough  as  the  computer-based 
training  lesson.  When  participants  referred  to  the  T.O.  during 
the  performance  test,  some  individuals  experienced  difficulty 
locating  and  following  the  directions.  In  contrast  to  the  T.O., 
the  CBT  was  viewed  as  "alive"  with  various  ways  of  emphasizing 
detail  and  critical  steps  in  performing  the  maintenance  tasks. 

The  participants'  perception  of  CBT  was  favorable.  They 
viewed  CBT  as  a  user-friendly  training  tool  that,  combined  with 
conventional  AF  OJT,  could  expedite  maintenance  task  training. 
Despite  the  participants'  certification  status,  they  felt  the  CBT 
added  to  their  knowledge.  IVD  was  preferred  over  CAI  because  of 
the  audio-video  presentation  of  the  task,  combined  with  the 
interactive  characteristics  of  the  training  lesson.  They  highly 
recommended  its  implementation  in  the  operational  setting  and 
felt  other  trainees  could  benefit  from  the  lessons. 
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This  study  is  not  without  the  inherent  limitations  of  field- 
based  research.  For  example,  there  was  no  opportunity  to 
investigate  the  psychometric  properties  (validity,  reliability) 
of  the  test  instruments  before  their  use.  However,  these 
instruments  were  developed  by  SMEs  and  validated  by  OJT 
supervisors.  Another  constraint  was  the  small  size  of  control 
and  treatment  groups. 

Finally,  we  were  not  able  to  control  the  assignment  of 
participants  to  either  the  treatment  or  control  groups. 
Consequently,  a  major  assumption  behind  this  study  was  that 
maintenance  personnel  were  randomly  assigned  to  their  respective 
units  by  the  AF  personnel  system.  This  assumption  is  critical  to 
our  research  design  and  necessary  to  ensure  equivalence  between 
test  groups,  but  may  not  be  true.  We  feel,  however,  that  this 
assumption  was  justified  given  the  apparent  equivalence  between 
our  test  groups  on  variables  such  as  time  in  service,  rank, 
number  of  times  a  task  was  performed,  and  time  since  last 
performance . 

Future  research  efforts  on  the  potential  of  CBT  in  the 
operational  setting  should  investigate  the  effects  of  human 
factor  variables.  Despite  the  benefits  derived  from  implementing 
this  training  technology,  it  will  not  be  realized  without  the 
acceptance  of  the  users.  Given  the  minimal  change  that  has 
occurred  in  AF  OJT  over  the  years,  acceptance  and  integration  of 
CBT  technology  may  not  be  easy.  Consequently,  additional 
research  on  AF  OJT  will  be  needed  to  heighten  our  awareness  of 
important  factors  conducive  to  implementing  this  technology  in 
the  operational  setting. 
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APPENDIX  A 


Analysis  of  Interview  Responses 

A  content  analysis  of  the  interview  responses  revealed  a 
favorable  perception  of  this  technology  in  the  work  setting.  As 
noted  on  Table  4,  respondents  felt  the  CBT  was  easy  to  use  and 
understand.  Eighty  percent  reported  that  the  CBT  was  helpful  in 
preparing  for  both  the  knowledge  and  performance  tests.  They  felt 
the  use  of  CBT  would  result  in  a  time  savings  and  provide  more 
organized  and  detailed  information.  In  addition,  they  felt  the 
performance  tests  provided  a  more  valid  measure  of  task 
proficiency. 


AOTS  PERFORMANCE  STUDY 
INTERVIEW 
N=15 


CBT - 

YES _ 

_ m 

Easy  to  operate? 

93% 

7% 

Easy  to  follow? 

86% 

14% 

Reduce  time  to  learn  task? 

87% 

13% 

Adequately  prepare  trainees  to 
perform  task? 

80% 

20% 

More  organized  information? 

87% 

13% 

Trainees  should  receive  more 
training  via  CBT? 

80% 

20% 

Are  performance  tests  better  for 
evaluating  task  proficiency  than 
current  OJT  method? 

71% 

29% 
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APPENDIX  B 

Sample  Performance  Evaluation  Checklist  and  Knowledge  Test 
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CAI  LESSON/TASK  ID/PEC/KNOWLEDGE  TEST  LISTING 


CAI  ID#-  42-1-1-1-1 

CAI  TITLE-  CSD/GEN  DOME  ASSY  INST. 

TASK  ID#-  L10012 

TASK  TITLE-  INSTALL  A  CONSTANT  SPEED  DRIVE/ GENERATOR  DOME 
OSR  CROSS-REFERENCE-  H00243 
PEC/OTG-  ATTACHED 
KNOWLEDGE  TEST-  ATTACHED 
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Item  203:  Oral  Test  Guide 


Initial  Author: 
Initial  Date: 

Last  Modified  by: 
Modify  Date: 


Haapanen,  Bob 
09  Dee  1987 
Haapanen,  Robert 
11  Jul  1988 


TERMINAL  BEHAVIOR  OBJECTIVE  :  INSTALL  A  CONSTANT  SPEED  DRIVE/GENERATOR  DOME 
IN  ACCORDANCE  WITH  TECHNICAL  ORDER  1F-4C-10. 


WEAPON  SYSTEM  :  RF-4C  and  F-40;  J79-15  ENG 

ACFT  CONFIG  :  N/A 

COMPONENT  CONFIG  :  N/A 

ESTIMATED  TIME  :  1  Hour 

TEAM  TASK  COOE  :  Not  a  tea*  task 
PREREQUISITE  TASK:  N/A 


TEN  for  EXAMINEE  :  CTK 

:  TORQUE  WRENCH  0-150  INCH  POUNDS 
:  TORQUE  WRENCH  0-300  INCH  POUNDS 


TEN  for  EVALUATOR:  Task  Checklist,  Pencil,  and  Clipboard 
EVALUATOR  INSTRUCTIONS: 
o  Acbiinister  the  evaluation  in  the  shop. 


o  During  the  evaluation  you  may  ask  the  examinee  to  explain  any  steps  that 
he/she  is  performing  and  you  are  unable  to  observe.  Simply  say  "TELL  ME  WHAT 
YOU  ARE  DOING  NOW."  Use  the  accompanying  checklist  to  record  the  examinee's 
actions  or  verbal  responses.  In  the  space  provided  to  the  right  of  each 
checklist  activity  number,  circle  Y  if  the  examinee  performed  the  step 
correctly,  circle  N  if  the  examinee  performed  the  step  incorrectly.  Circle  U 
if  the  examinee  was  unable  to  perform  the  step  due  to  conditions  outside  of 
his  of  her  control  or  you  were  unable  to  observe  the  step  for  some  other 
reason. 

o  Read  the  following  instructions  slowly  and  clearly  to  the  examinee. 
Substitute  the  word(s)  in  parenthesees  as  requried.  Emphasize  that  the 
examinee  should  perform  steps  alone,  but  can  have  assistance  when  necessary, 
and  that  the  examinee  is  to  verbally  tell  you  when  he/she  makes  s  visual 
check. 


SAY  TO  THE  EXAMINEE:  INSTALL  A  CONSTANT  SPEED  DRIVE/GENERATOR  DOME  IN 
ACCORDANCE  WITH  TECHNICAL  ORDER  1F-4C-10.  Obtain  all  tools,  materials,  and 
equipment  before  starting  the  task.  Psrform  all  steps  alone  except  when  to  do 
so  might  result  in  injury  to  personnel  or  damage  to  equipment.  TELL  ME  IF  YOU 
DEVIATE  FROM  THE  TECHNICAL  ORDER  SUCH  AS  PERFORMING  A  STEP  OUT  OF  SEQUENCE. 
TELL  ME  WHEN  YOU  PERFORM  A  STEP  OR  CHECK  THAT  I  CAN  NOT  SEE  YOU  PERFORMING, 
SUCH  AS  ENSURING  THAT  SWITCHES  ARE  IN  THE  CORRECT  POSITION  OR  THAT  SAFETY 
DEVICES  ARE  INSTALLED.  WHEN  YOU  MAKE  A  CHECK,  TELL  ME  EXACTLY  WHAT  YOU  ARE 
LOOKING  FOR  AN0  WHAT  YOU  FINO. 

Tell  me  when  you  are  ready  to  begin  the  tssk  and  when  you  have  completed  the 
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task. 


DID  THE  EXAMINEE  PERFORM  OR  SAY  HE/SHE  SHOULD  PERFORM  THE  FOLLOWING 
ACTIONS: 


Item  148:  Performance  Checklist 

1  Y  N  U  Safety  the  screw  securing  the  bracket  (14)  on  the  forward  left  side 

of  the  generator  with  .032  loekwire?  (20KVA  only) 

2  Y  n  i  .nsure  eeals  pert  nusbers  53-50031-7  and  53-50017-63  were 

serviceable? 

3  Y  n  U  Carefully  feed  the  electrical  leads  through  the  CSD  dome  to  avoid 

damaging  the  terminal  block  insulator  dividers? 

4  Y  N  U  Secure  electrical  lead  X75  to  terminal  G? 

5  Y  N  U  Secure  electrical  lead  XS5  to  terminal  T1? 

6  Y  N  U  Secure  electrical  lead  X56  to  terminal  T2? 

7  Y  n  U  Secure  electrical  lead  X57  to  terminal  T3? 

8  Y  n  u  Secure  electrical  lead  X58  to  terminal  F? 

9  Y  n  u  Secure  electrical  lead  X65  to  terminal  P? 

10  y  n  u  Ensure  the  flange  on  the  aft  and  of  the  dome  was  inserted  into  the 

split  of  the  dome  locking  ring? 

11  y  n  u  Secure  the  left  dome  strut  to  the  air  oil  cooler  pad? 

12  Y  N  u  Ensure  the  Permacel  tape  on  the  left  dams  strut  was  not  damaged 

during  the  installation? 

13  Y  M  U  Secure  the  right  dome  strut  to  the  air  oil  cooler  pad? 

14  Y  N  u  Ensure  the  Permacel  tape  on  the  right  dome  strut  was  not  tismsgsd 

during  the  installation? 
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15  Y  N  U  Torque  the  bolts  securing  the  dame  struts  (6  end  15)  to  70  inch 

pounds? 

16  Y  N  U  Safety  the  bolts  securing  the  dome  struts  with  .032  lockwire? 

17  Y  N  U  Ensure  the  mismatch  between  the  leading  edge  of  the  strut  end  the 

eiroil  cooler  strut  pads  were  less  then  .090  inch? 

18  Y  N  U  Ensure  the  mismatch  between  the  sides  of  the  strut  end  door 

assemblies  the  mold  line  of  the  strut  pads  were  less  then  .050  inch? 

19  Y  N  U  Position  the  electrical  leads  to  provide  maxinun  clearance  between 

the  leads  and  the  dome  structure? 

20  Y  N  U  Torque  the  nuts  securing  the  lead's  terminals  to  60-85  inch  pounds? 

21  Y  N  U  Connect  electrical  connector  3P616  to  the  eft  end  of  the  CSO? 

22  Y  N  U  Safety  the  screw  (16)  securing  the  CSO  electrical  lead  to  CSD  with 

.032  lockwire  (20  KVA  only)? 

23  Y  N  U  Safety  the  three  bolts  (8)  securing  the  CSO  electrical  lead  to  the 

CSO  with  .032  lockwire  (30  KVA  only)? 

24  Y  N  U  Install  the  seal  (2)  on  the  forward  end  of  the  anti  icing  line  with 

the  open  face  of  the  saal  positioned  forward? 

25  Y  »  ll  Install  a  seal  on  between  the  aft  and  of  the  anti -icing  line  and  the 

C SO/GEN  forward  coupling? 

26  r  N  U  lubricate  the  bolts  securing  the  anti -icing  line  to  the  forward 

coupling? 

27  Y  N  U  Torque  the  bolts  securing  the  anti-icing  line  to  the  forward 

coupling  to  250-270  inch  pounds? 

28  Y  N  U  Safety  the  bolts  securing  the  anti -icing  line  to  the  forward 

coupling  with  .032  lockwire? 

29  Y  N  U  Lubricate  the  inner  surface  of  bushing  (12)  with  W-P-236? 

30  Y  N  U  Insert  the  bushing  (12)  into  dome  with  the  split  facing  aft? 


3.  Item  1073:  True  false  Question 


The  protective  cape  and  plugs  are  removed  before  the  CSO  dome  assembly  is 
installed. 


a.  True 

b.  false 


Correct  response  is:  A 
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feedback  for  incorrect  response: 

None  specified. 


'*••**•****•••*••*•••«•««*«**•«««***••*•«*•***••*••**••*••••«*••*•*•* 

4.  Item  1074:  True  False  Question 

The  nutplate  bracket  is  installed  onto  the  20  KVA  Generator  before  the 
dome  assembly  in  installed. 

a.  True 

b.  False 


Correct  response  is:  A 


Feedback  for  incorrect  response: 
None  specified. 


5.  Item  1075:  True  False  Question 


To  prevent  damage  to  the  generator  electrical  connector  it  is  recommended 
that  you  tape  the  generetor  leade  together  before  routing  the  leade 
through  the  dome. 


a.  True 

b.  False 


Correct  response  is:  A 


Feedback  for  incorrect  reaponee: 
None  specified. 


6.  Item  1078:  True  False  Question 

Color  coding  is  used  to  identify  which  electrical  lead  connects  to  which 
generator  terminal  post. 

a.  True 

b.  False 


Correct  response  is:  I 


Feedback  for  incorrect  response: 
None  specified. 
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7.  Item  1082:  True  Falsa  Question 

The  electrical  leads  may  touch  the  dome  assembly  only  if  the  leads  have 
been  properly  insulated. 

a.  True 

b.  False 


Correct  response  is:  8 


Feedback  for  incorrect  responae: 
None  specified. 


8.  Item  1085:  True  False  Question 

A  bushing  in  positioned  around  the  electrical  lead  in  the  left  strut  area 
to  prevent  chafing. 

a.  True 

b.  False 


Correct  response  is:  A 


Feedback  for  incorrect  response: 
None  specified. 


9.  Item  1087:  True  False  Question 


Damaged  threads  on  the  CSC  oil  supply  port  Mill  require  the  CSC  to  be 
replaced. 


a.  True 

b.  False 


Correct  response  is:  • 


Feedback  for  incorrect  response: 
None  specified. 
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10.  Iteffi  1078:  Multiple  Choice  Question 

The  nutplate  bracket  secures  the 

A.  CSO  electrical  lead  to  the  generator. 

B.  engine  electrical  harness  to  the  dome  assembly. 

C.  anti -icing  line  to  the  generator. 

0.  CSO  oil  drain  line  to  the  CSO  unit. 


Correct  response  is:  A 


Feedback  for  alternative  A 
Correct. 

Feedback  for  alternative  B 

Incorrect,  it  secures  the  CSO  electrical  lead  to  the  generator. 
Feedback  for  alternative  C 

Incorrect,  it  secures  the  CSO  electrical  lead  to  the  generator. 
Feedback  for  alternative  0 

incorrect,  it  secures  the  CSO  electrical  lead  to  the  generator. 


11.  I  taw  1077:  Multiple  Choice  Question 


What  engine  component,  by  design,  provides  the  dome  strut  Mounting  peds? 

A.  The  air-oil  cooler. 

B.  Engine  front  frame. 

C.  Compressor  case. 

D.  Transfer  gearbox. 


Correct  response  is:  A 


Feedback  for  alternative  A 
Correct. 


Feedback  for  alternative  0 

Incorrect,  the  air-oil  cooler 

Feedback  for  alternative  C 

Incorrect,  the  air-oil  cooler 

Feedback  for  alternative  0 

Incorrect,  the  air-oil  cooler 


provides  the  dome 
provides  the  dome 
provides  the  dome 


strut  mounting  pads, 
strut  moulting  pads, 
strut  mounting  pods. 
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12.  Item  1079:  Multiple  Choice  Question 

whet  engine  component  MUST  have  been  installed  prior  to  installing  the 
CSO  dome  assembly? 

A.  The  CSO  dome  locking  ring  and  the  air-oil  cooler  assembly. 

B.  The  generator  electrical  harnesa  and  the  air-oil  cooler  asseebly. 

C.  CSO  oil  lines  and  the  generator  electrical  leada. 

0.  CSO  oil  lines  and  the  CSO  dome  locking  ring. 


Correct  response  is:  A 


Feedback  for  alternative  A 
That's  correct 

Feedback  for  alternative  B 

Incorrect,  its  the  CSO  dome  locking  ring  and  the  air-oil  cooler  assembly. 
Feedback  for  alternative  C 

Incorrect,  its  the  CSO  dome  locking  ring  and  the  air-oil  cooler  assembly. 
Feedback  for  alternative  0 

Incorrect,  its  the  CSO  dome  locking  ring  and  the  air-oil  cooler  assembly. 


13.  Item  1080:  Multiple  Choice  Question 


why  must  the  CSO  dome  struts  be  properly  aligned  with  the  air-oil  cooler 
strut  pads? 


A.  To  insure  clearance  between  the  struts  ard  the  CSO  electrical  leads 
and  the  oil  lines. 

B.  To  allow  installation  of  the  dome  assembly  to  the  air-oil  cooler 
mount  bolts. 

C.  To  prevent  chafing  between  the  generator  and  the  electrical  leads. 


Correct  response  is:  A 


Feedback  for  alternative  A 
Correct. 

Feedback  for  alternative  B 

Incorrect,  it  is  to  insure  clearance  between  the  struts  and  the  CSO 
electrical  leads  and  the  oil  lines. 

Feedback  for  alternative  C 

Incorrect,  it  is  to  insure  clearance  between  the  struts  and  the  CSO 
electrical  leads  and  the  oil  lines. 
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14.  Item  1001:  Multiple  Choice  Question 

whet  tool  is  used  to  perform  the  dome  strut  to  the  sir-oil  cooler  ped 
alignment  check? 

A.  Thickness  gsge. 

8.  Straight  edge  ruler. 

C.  Depth  micrometer. 

D.  Outside  micrometer. 


Correct  response  is:  A 


Feedback  for  alternative  A 
Correct 

Feedback  for  alternative  • 

Incorrect,  the  thickness  gage  is  used  to  perform  the  dome  strut  to  the 
air-oil  cooler  ped  alignment  check. 

Feedback  for  alternative  C 

Incorrect,  the  thickness  gage  is  used  to  perform  the  dome  strut  to  the 
sir-oil  cooler  pad  alignment  check. 

Feedback  for  alternative  0 

Incorrect,  the  thickness  gage  is  used  to  perform  the  dome  strut  to  the 
air-oil  cooler  pad  alignment  check. 


15.  Item  1083:  Multiple  Choice  Question 


How  is  the  forward  anti -icing  line  seal  positioned? 


A.  with  the  open  face  of  the  seal  forward. 

8.  With  the  open  face  of  the  seal  aft. 

C.  Inside  the  CSO  dome  receptacle. 

c.  on  the  forward  and  of  the  anti -icing  line. 


Correct  response  is:  A 


Feedback  for  alternative  A 
Very  Good. 

Feedback  for  alternative  • 

Incorrect,  it  is  positioned  with  the  open  face  of  the  seat  forward. 
Feedback  for  alternative  C 

Incorrect,  it  is  positioned  with  the  open  face  of  the  seal  forward. 
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Paadback  for  altarnativa  D 

incorract,  it  i«  positionad  with  tha  opan  faea  of  tha  saal  forward. 


16.  I  taw  1084:  Hultipla  Choica  Quastion 

,B  U*#d  00  th«  •**«•*«<"■  l «owit  bolts  during 
installation?  * 

A.  Mil- 1-25681 . 

B.  MIl-l-7808. 

C.  Patrolatun. 

0.  Moly  Grada  3  graasa. 


Corract  raaponaa  it:  A 


Paadback  for  altarnativa  A 
Vary  Good. 

Paadback  for  altarnativa  8 

Paadback  for  altarnativa  C 

Jss'scuJuSti.** "  '*•  >•» 

Faadback  for  altarnativa  0 


17.  I  taw  1086:  Hultipla  Choica  Quastion 

Which  conponant  pasaas  through  tha  right  CSD  dais  strut? 

A.  Tha  CSD  oil  supply  and  ratum  Unas. 

8.  Tha  CSD  oil  supply  and  drain  linas. 

C.  Tha  CSO/GEN  alactrical  hamass. 

D.  Tha  CSD  oil  ratum  and  drain  linas. 


Corract  raaponaa  is:  A 


P*«**ck  for  altarnativa  A 
Corract. 

Paadback  for  altarnativa  8 

Mo.  Tha  CSO  oil  supply  and  ratum  linos  posa  through  tha  right  CSD 
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feedback  for  alternative  C 

No.  The  CSO  oil  supply  and  return  lines  pass  through  the  right  CSO  dome 
strut. 

feedback  for  alternative  0 

no.  The  CSO  oil  supply  and  return  lines  pass  through  the  right  CSO  dome 
strut. 


18.  Item  1088:  Multiple  Choice  Ouestion 

What  is  placed  on  the  CSO  oil  supply  and  return  lines  to  prevent  chafing. 


A.  Chafe  guard  (spril  wrap). 

B.  Spacer  between  oil  lines. 

C.  Metel  chafe  blenket. 

0.  Adel  clamps. 


Correct  response  is:  A 


Feedback  for  alternative  A 
Very  Good. 

Feedback  for  alternative  B 

Incorrect.  The  chafe  guard  (spril  wrap)  is  placed  on  the  CSO  oil  supply  and 
return  lines  to  prevent  chafing. 

Feedback  for  alternative  C 

Incorrect.  The  chafe  guard  (spril  wrap)  ia  placed  on  the  CSO  oil  supply  and 
return  lines  to  prevent  chafing. 

Feedback  for  alternative  0 

Incorrect.  The  chafe  guard  (spril  wrap)  is  placed  on  the  CSO  oil  sipply  and 
return  lines  to  prevent  chafing. 


19.  Item  1089:  Multiple  Choice  Ouestion 

What  c exponents  pass  through  the  left  dome  strut? 

A.  The  electrical  lead  and  the  CSO  oil  drain  line. 

B.  The  electrical  lead  and  the  CSO  oil  supply  line. 

C.  The  electrical  lead  and  the  CSO  oil  return  line. 

0.  The  CSO  oil  supply  and  oil  return  lines. 


Correct  response  is:  A 

Feedback  for  alternative  A 
Very  Good. 
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Feedback  for  alternative  8 

Incorrect.  The  electrical  lead  and  CSO  oil  drain  line  peases  through  the 
left  done  strut. 

Feedback  for  alternative  C 

Incorrect.  The  electrical  lead  and  CSO  oil  drain  line  passes  through  the 
left  done  strut. 

Feedback  for  alternative  0 

Incorrect.  The  electrical  lead  and  CSO  oil  drain  line  passes  through  the 
left  dome  strut. 


20.  Item  1090:  Multiple  Choice  Question 

How  is  the  CSO  drain  hose  secured  to  the  CSO? 

A.  By  the  oil  drain  fitting. 

S.  By  the  oil  drain  reducer. 

C.  By  a  fitting  extending  from  the  CSO  filter. 
0.  By  a  fitting  on  the  CSO  housing. 


Correct  response  is:  A 


Feedback  for  alternative  A 
very  Good. 

Feedback  for  alternative  B 

Incorrect.  The  oil  drain  fitting  secures  the  CSO  drain  hose  to  the  CSO. 
Feedback  for  alternative  C 

Incorrect.  The  oil  drain  fitting  secures  the  CSO  drain  hose  to  the  CSO. 
Feedback  for  alternative  0 

Incorrect.  The  oil  drain  fitting  secures  the  CSO  drain  hose  to  the  CSO. 


21.  Item  1091:  Multiple  Choice  Question 

tftat  i teals)  are  installed  to  prevent  chafing  of  the  electrical  harness 
and  the  CSO  oil  drain  line? 

A.  Spacers  and  seal. 

B.  Adel  claapa. 

C.  Chafe  guard. 

0.  Spacers  only. 


Correct  response  is:  A 

Feedback  for  alternative  A 
Very  Good. 
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feedback  for  altarnativa  B 

Incorrect.  A  seal  and  spacers  are  installed  to  prevent  chafing  of  the 
electrical  harness  and  the  CSO  oil  drain  line. 

Feedback  for  alternative  C 

Incorrect.  A  seal  and  spacers  are  installed  to  prevent  chafing  of  the 
electrical  harness  and  the  CSD  oil  drain  line. 

Feedback  for  alternative  0 

Incorrect.  A  seal  and  spacers  are  installed  to  prevent  chafing  of  the 
electrical  harness  and  the  CSO  oil  drain  line. 
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